﻿Prac-4  [Cloud DataFlow Pipeline]


1. Go to Dataflow in searchbar
2. Select Jobs        
3. Enable Dataflow and related APIs
        3.1. Select FIX button for all 
        3.2. Wait till done
        3.3. Select Get Started
4. google cloud shell - local
4.0. Install Gcloud CLI installer

gcloud init
gcloud config set project ( project name )
gcloud auth application-default login

        4.3.1. select all options while authentication

pip install apache-beam[gcp]

gcloud storage buckets create gs://cloud-dataflow-pipeline --default-storage-class=STANDARD


python -m apache_beam.examples.wordcount --runner DataflowRunner --project ( project name ) --region asia-south2 --input gs://dataflow-samples/shakespeare/kinglear.txt --output gs://(bucket name)/results/outputs --temp_location gs://( bucket name )/tmp


6. Check Jobs sections for success message 
6.1. keep cmd running till success takes 5-10mins
7. View output in cloud storage bucket
        7.1. Check Authenticated URL for output